<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  
  <title>爬取自己CSDN上所有的博客并转存为md文件 | 蓝湖畔淅淅沥沥的雨</title>
  <meta name="viewport" content="width=device-width, initial-scale=1, maximum-scale=1">
  <meta name="description" content="1.先考虑怎么获得自己所有博客的地址 a.打开自己的个人博客，尝试从HTML里提取有用信息，没能成功。 b.发现下滑后动态加载自己的博客，于是抓包，找到连接  c.发现连接中除了page&#x3D;%d这一部分外，其余都相同，所以考虑使用for循环请求多个连接，并在返回数据中提取网址 1234567891011121314151617import requestsimport reN &#x3D; 6idname">
<meta property="og:type" content="article">
<meta property="og:title" content="爬取自己CSDN上所有的博客并转存为md文件">
<meta property="og:url" content="http://example.com/2022/05/06/%E7%88%AC%E5%8F%96%E8%87%AA%E5%B7%B1CSDN%E4%B8%8A%E6%89%80%E6%9C%89%E7%9A%84%E5%8D%9A%E5%AE%A2%E5%B9%B6%E8%BD%AC%E5%AD%98%E4%B8%BAmd%E6%96%87%E4%BB%B6/index.html">
<meta property="og:site_name" content="蓝湖畔淅淅沥沥的雨">
<meta property="og:description" content="1.先考虑怎么获得自己所有博客的地址 a.打开自己的个人博客，尝试从HTML里提取有用信息，没能成功。 b.发现下滑后动态加载自己的博客，于是抓包，找到连接  c.发现连接中除了page&#x3D;%d这一部分外，其余都相同，所以考虑使用for循环请求多个连接，并在返回数据中提取网址 1234567891011121314151617import requestsimport reN &#x3D; 6idname">
<meta property="og:locale" content="zh_CN">
<meta property="og:image" content="http://example.com/2022/05/06/%E7%88%AC%E5%8F%96%E8%87%AA%E5%B7%B1CSDN%E4%B8%8A%E6%89%80%E6%9C%89%E7%9A%84%E5%8D%9A%E5%AE%A2%E5%B9%B6%E8%BD%AC%E5%AD%98%E4%B8%BAmd%E6%96%87%E4%BB%B6/zhuabaolianjie.png">
<meta property="article:published_time" content="2022-05-06T09:59:32.000Z">
<meta property="article:modified_time" content="2022-05-06T10:30:37.077Z">
<meta property="article:author" content="StarsWhisper">
<meta property="article:tag" content="爬虫">
<meta name="twitter:card" content="summary">
<meta name="twitter:image" content="http://example.com/2022/05/06/%E7%88%AC%E5%8F%96%E8%87%AA%E5%B7%B1CSDN%E4%B8%8A%E6%89%80%E6%9C%89%E7%9A%84%E5%8D%9A%E5%AE%A2%E5%B9%B6%E8%BD%AC%E5%AD%98%E4%B8%BAmd%E6%96%87%E4%BB%B6/zhuabaolianjie.png">
  
    <link rel="alternate" href="/atom.xml" title="蓝湖畔淅淅沥沥的雨" type="application/atom+xml">
  
  
    <link rel="icon" href="/favicon.png">
  
  
    <link href="//fonts.googleapis.com/css?family=Source+Code+Pro" rel="stylesheet" type="text/css">
  
  
<link rel="stylesheet" href="/css/style.css">

  
<link rel="stylesheet" href="/plugin/bganimation/bg.css">

  

  <link href="https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.10.0/katex.min.css" rel="stylesheet" type="text/css">
<meta name="generator" content="Hexo 6.1.0"></head>

<body>
  <div id="container">
    <div id="wrap">
      <div class="outer">
        <div class="widget-wrap mobile-header">
  <h3 class="widget-title"></h3>
  <div class="widget">
    <img class="avatar" src="/images/avatar.png">
    <h2 class="author">StarsWhisper</h2>
    <h3 class="description"></h3>
    <div class="count-box">
      <a href="/archives"><div><strong>75</strong><br>文章</div></a>
      <a href="/categories"><div><strong>31</strong><br>分类</div></a>
      <a href="/tags"><div><strong>56</strong><br>标签</div></a>
    </div>
    <ul class="blog-link">
     
          <a href="/" title="Home">
            <li>主页</li>
          </a>
        
          <a href="/archives" title="Archives">
            <li>归档</li>
          </a>
        
          <a href="/categories" title="Categories">
            <li>分类</li>
          </a>
        
          <a href="/tags" title="Tags">
            <li>标签</li>
          </a>
        
          <a href="/knightabout" title="Knightabout">
            <li>关于</li>
          </a>
        
          <a href="/bridges" title="Bridges">
            <li>传送门</li>
          </a>
        
          <a href="/announcement" title="Announcement">
            <li>公告</li>
          </a>
        
    </ul>
  </div>
</div>

        <section id="main"><article id="post-爬取自己CSDN上所有的博客并转存为md文件" class="wow slideInRight article article-type-post" itemscope itemprop="blogPost">
  <div class="article-meta">
    <a href="/2022/05/06/%E7%88%AC%E5%8F%96%E8%87%AA%E5%B7%B1CSDN%E4%B8%8A%E6%89%80%E6%9C%89%E7%9A%84%E5%8D%9A%E5%AE%A2%E5%B9%B6%E8%BD%AC%E5%AD%98%E4%B8%BAmd%E6%96%87%E4%BB%B6/" class="article-date">
  <time class="post-time" datetime="2022-05-06T09:59:32.000Z" itemprop="datePublished">
    <span class="post-month">5月</span><br/>
    <span class="post-day">06</span>
  </time>
</a>
   
  </div>
  <div class="article-inner">
    
    
      <header class="article-header">
        
  
    <h1 class="article-title" itemprop="name">
      爬取自己CSDN上所有的博客并转存为md文件
    </h1>
  

        <div>
          
  <div class="article-category">
    <a class="article-category-link" href="/categories/%E6%95%99%E7%BB%83%E6%88%91%E6%83%B3%E5%AD%A6%E6%8C%82%E8%BE%B9%E8%BA%B2%E7%89%9B/">教练我想学挂边躲牛</a>,<a class="article-category-link" href="/categories/%E6%95%99%E7%BB%83%E6%88%91%E6%83%B3%E5%AD%A6%E6%8C%82%E8%BE%B9%E8%BA%B2%E7%89%9B/%E7%88%AC%E8%99%AB/">爬虫</a>
  </div>

          
              

          
        </div>
      </header>
    
    <div class="article-entry" itemprop="articleBody">
      
        <h2 id="1先考虑怎么获得自己所有博客的地址"><a class="markdownIt-Anchor" href="#1先考虑怎么获得自己所有博客的地址"></a> 1.先考虑怎么获得自己所有博客的地址</h2>
<p>a.打开自己的个人博客，尝试从HTML里提取有用信息，没能成功。</p>
<p>b.发现下滑后动态加载自己的博客，于是抓包，找到连接</p>
<img src="/2022/05/06/%E7%88%AC%E5%8F%96%E8%87%AA%E5%B7%B1CSDN%E4%B8%8A%E6%89%80%E6%9C%89%E7%9A%84%E5%8D%9A%E5%AE%A2%E5%B9%B6%E8%BD%AC%E5%AD%98%E4%B8%BAmd%E6%96%87%E4%BB%B6/zhuabaolianjie.png" class="" title="zhuabaolianjie">
<p>c.发现连接中除了page=%d这一部分外，其余都相同，所以考虑使用for循环请求多个连接，并在返回数据中提取网址</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">import</span> re</span><br><span class="line"></span><br><span class="line">N = <span class="number">6</span></span><br><span class="line">idname = <span class="string">&#x27;wldcmzy&#x27;</span></span><br><span class="line"></span><br><span class="line">headers = &#123;</span><br><span class="line">    <span class="string">&#x27;User-Agent&#x27;</span>: <span class="string">&#x27;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.41 Safari/537.36 Edg/101.0.1210.32&#x27;</span></span><br><span class="line">&#125;</span><br><span class="line">url_base = <span class="string">&#x27;https://blog.csdn.net/community/home-api/v1/get-business-list?page=&#123;page&#125;&amp;size=20&amp;businessType=lately&amp;noMore=false&amp;username=&#123;name&#125;&#x27;</span>.<span class="built_in">format</span>(page = <span class="string">&#x27;&#123;page&#125;&#x27;</span>, name = idname)</span><br><span class="line"></span><br><span class="line">lst = []</span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">1</span>, N):</span><br><span class="line">    res = requests.get(url_base.<span class="built_in">format</span>(page = i), headers = headers)</span><br><span class="line">    lst.extend(re.findall(<span class="string">&quot;https://blog.csdn.net/%s/article/details/[0-9]+&quot;</span> % idname, res.text))</span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="2写抓取博文的代码"><a class="markdownIt-Anchor" href="#2写抓取博文的代码"></a> 2.写抓取博文的代码</h2>
<p>由于本人太菜了，没有成功吧博文提取出来，所以去网上搜索到了<a target="_blank" rel="noopener" href="https://blog.csdn.net/haojie_duan/article/details/122864466">一篇大佬的文章</a>，这篇文章是爬取CSDN热门文章的，可以复制源码略加改动，打到爬自己博文的目的</p>
<p>中间还是遇到了一些错误的，毕竟自己的需求和原文有所不同，比如大佬的代码里没有处理特殊字符\t，以及使用BeautifSoup检索标签时需要在class_参数中加上&quot;markdown_views prism-atom-one-dark&quot;。这些问题可以很快被发现并且修正。</p>
<p>附上改后的代码:</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br><span class="line">67</span><br><span class="line">68</span><br><span class="line">69</span><br><span class="line">70</span><br><span class="line">71</span><br><span class="line">72</span><br><span class="line">73</span><br><span class="line">74</span><br><span class="line">75</span><br><span class="line">76</span><br><span class="line">77</span><br><span class="line">78</span><br><span class="line">79</span><br><span class="line">80</span><br><span class="line">81</span><br><span class="line">82</span><br><span class="line">83</span><br><span class="line">84</span><br><span class="line">85</span><br><span class="line">86</span><br><span class="line">87</span><br><span class="line">88</span><br><span class="line">89</span><br><span class="line">90</span><br><span class="line">91</span><br><span class="line">92</span><br><span class="line">93</span><br><span class="line">94</span><br><span class="line">95</span><br><span class="line">96</span><br><span class="line">97</span><br><span class="line">98</span><br><span class="line">99</span><br><span class="line">100</span><br><span class="line">101</span><br><span class="line">102</span><br><span class="line">103</span><br><span class="line">104</span><br><span class="line">105</span><br><span class="line">106</span><br><span class="line">107</span><br><span class="line">108</span><br><span class="line">109</span><br><span class="line">110</span><br><span class="line">111</span><br><span class="line">112</span><br><span class="line">113</span><br><span class="line">114</span><br><span class="line">115</span><br><span class="line">116</span><br><span class="line">117</span><br><span class="line">118</span><br><span class="line">119</span><br><span class="line">120</span><br><span class="line">121</span><br><span class="line">122</span><br><span class="line">123</span><br><span class="line">124</span><br><span class="line">125</span><br><span class="line">126</span><br><span class="line">127</span><br><span class="line">128</span><br><span class="line">129</span><br><span class="line">130</span><br><span class="line">131</span><br><span class="line">132</span><br><span class="line">133</span><br><span class="line">134</span><br><span class="line">135</span><br><span class="line">136</span><br><span class="line">137</span><br><span class="line">138</span><br><span class="line">139</span><br><span class="line">140</span><br><span class="line">141</span><br><span class="line">142</span><br><span class="line">143</span><br><span class="line">144</span><br><span class="line">145</span><br><span class="line">146</span><br><span class="line">147</span><br><span class="line">148</span><br><span class="line">149</span><br><span class="line">150</span><br><span class="line">151</span><br><span class="line">152</span><br><span class="line">153</span><br><span class="line">154</span><br><span class="line">155</span><br></pre></td><td class="code"><pre><span class="line"><span class="string">&quot;&quot;&quot;</span></span><br><span class="line"><span class="string">@Author:survive</span></span><br><span class="line"><span class="string">@Blog(个人博客地址): https://blog.csdn.net/haojie_duan</span></span><br><span class="line"><span class="string"> </span></span><br><span class="line"><span class="string">@File:csdn.py.py</span></span><br><span class="line"><span class="string">@Time:2022/2/10 8:49</span></span><br><span class="line"><span class="string"> </span></span><br><span class="line"><span class="string">@Motto:我不知道将去何方，但我已在路上。——宫崎骏《千与千寻》</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">代码思路：</span></span><br><span class="line"><span class="string">1.确定目标需求:将csdn文章内容保存成 html、PDF、md格式</span></span><br><span class="line"><span class="string">    - 1.1首先保存为html格式：获取列表页中所有的文章ur1地址，请求文章ur1地址获取我们需要的文章内容</span></span><br><span class="line"><span class="string">    - 1.2 通过 wkhtmitopdf.exe把html文件转换成PDF文件</span></span><br><span class="line"><span class="string">    - 1.3 通过 wkhtmitopdf.exe把html文件转换成md文件</span></span><br><span class="line"><span class="string"></span></span><br><span class="line"><span class="string">2.请求ur1获取网页源代码</span></span><br><span class="line"><span class="string">3.解析数据，提取自己想要内容</span></span><br><span class="line"><span class="string">4.保存数据</span></span><br><span class="line"><span class="string">5.转换数据类型把HTML转换成PDF、md文伴</span></span><br><span class="line"><span class="string">&quot;&quot;&quot;</span></span><br><span class="line"></span><br><span class="line"></span><br><span class="line">html_str = <span class="string">&quot;&quot;&quot;</span></span><br><span class="line"><span class="string">&lt;!DOCTYPE html&gt;</span></span><br><span class="line"><span class="string">&lt;html lang=&quot;en&quot;&gt;</span></span><br><span class="line"><span class="string">&lt;head&gt;</span></span><br><span class="line"><span class="string">    &lt;meta charset=&quot;UTF-8&quot;&gt;</span></span><br><span class="line"><span class="string">    &lt;title&gt;Document&lt;/title&gt;</span></span><br><span class="line"><span class="string">&lt;/head&gt;</span></span><br><span class="line"><span class="string">&lt;body&gt;</span></span><br><span class="line"><span class="string">&#123;article&#125;</span></span><br><span class="line"><span class="string">&lt;/body&gt;</span></span><br><span class="line"><span class="string">&lt;/html&gt;</span></span><br><span class="line"><span class="string">&quot;&quot;&quot;</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"><span class="keyword">import</span> parsel</span><br><span class="line"><span class="comment">#import pdfkit   #用来将html转为pdf</span></span><br><span class="line"><span class="keyword">import</span> re</span><br><span class="line"><span class="keyword">import</span> os</span><br><span class="line"><span class="keyword">import</span> urllib.parse</span><br><span class="line"><span class="keyword">from</span> bs4 <span class="keyword">import</span> BeautifulSoup</span><br><span class="line"><span class="keyword">import</span> html2text    <span class="comment">#用来将html转换为md</span></span><br><span class="line"><span class="keyword">import</span> random</span><br><span class="line"></span><br><span class="line"><span class="comment"># user_agent库：每次执行一次访问随机选取一个 user_agent，防止过于频繁访问被禁止</span></span><br><span class="line">USER_AGENT_LIST = [</span><br><span class="line">    <span class="string">&quot;Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1207.1 Safari/537.1&quot;</span>,</span><br><span class="line">    <span class="string">&quot;Mozilla/5.0 (X11; CrOS i686 2268.111.0) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11&quot;</span>,</span><br><span class="line">    <span class="string">&quot;Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1092.0 Safari/536.6&quot;</span>,</span><br><span class="line">    <span class="string">&quot;Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.6 (KHTML, like Gecko) Chrome/20.0.1090.0 Safari/536.6&quot;</span>,</span><br><span class="line">    <span class="string">&quot;Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; 360SE)&quot;</span>,</span><br><span class="line">    <span class="string">&quot;Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3&quot;</span>,</span><br><span class="line">    <span class="string">&quot;Mozilla/5.0 (Windows NT 6.1) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.1 Safari/536.3&quot;</span>,</span><br><span class="line">    <span class="string">&quot;Mozilla/5.0 (Windows NT 6.2) AppleWebKit/536.3 (KHTML, like Gecko) Chrome/19.0.1061.0 Safari/536.3&quot;</span>,</span><br><span class="line">    <span class="string">&quot;Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24&quot;</span>,</span><br><span class="line">    <span class="string">&quot;Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/535.24 (KHTML, like Gecko) Chrome/19.0.1055.1 Safari/535.24&quot;</span></span><br><span class="line">    <span class="string">&quot;Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36&quot;</span></span><br><span class="line">]</span><br><span class="line"></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">CSDNSpider</span>():</span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self, lst</span>):</span><br><span class="line">        self.url = <span class="string">&#x27;https://blog.csdn.net/csdndevelopers/category_10594816.html&#x27;</span></span><br><span class="line">        self.headers = &#123;</span><br><span class="line">            <span class="string">&#x27;user-agent&#x27;</span>:random.choice(USER_AGENT_LIST)</span><br><span class="line">        &#125;</span><br><span class="line">        </span><br><span class="line">        self.lst = lst</span><br><span class="line"></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">send_request</span>(<span class="params">self, url</span>):</span><br><span class="line">        response = requests.get(url=url, headers=self.headers)</span><br><span class="line">        response.encoding = <span class="string">&quot;utf-8&quot;</span></span><br><span class="line">        <span class="keyword">if</span> response.status_code == <span class="number">200</span>:</span><br><span class="line">            <span class="keyword">return</span> response</span><br><span class="line"></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">parse_content</span>(<span class="params">self, reponse</span>):</span><br><span class="line">        html = reponse.text</span><br><span class="line">        selector = parsel.Selector(html)</span><br><span class="line">        href = selector.css(<span class="string">&#x27;.column_article_list a::attr(href)&#x27;</span>).getall()</span><br><span class="line">        name = <span class="number">00</span></span><br><span class="line">        <span class="keyword">for</span> link <span class="keyword">in</span> href:</span><br><span class="line">            <span class="built_in">print</span>(link)</span><br><span class="line">            name = name + <span class="number">1</span></span><br><span class="line">            <span class="comment"># 对文章的url地址发送请求</span></span><br><span class="line">            response = self.send_request(link)</span><br><span class="line">            <span class="keyword">if</span> response:</span><br><span class="line">                self.parse_detail(response, name)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">parse_detail</span>(<span class="params">self, response, name</span>):</span><br><span class="line">        html = response.text</span><br><span class="line">        <span class="comment"># print(html)</span></span><br><span class="line">        selector = parsel.Selector(html)</span><br><span class="line">        title = selector.css(<span class="string">&#x27;#articleContentId::text&#x27;</span>).get()</span><br><span class="line">        <span class="comment"># content = selector.css(&#x27;#content_views&#x27;).get()</span></span><br><span class="line"></span><br><span class="line">        <span class="comment"># 由于这里使用parsel拿到的网页文件，打开会自动跳转到csdn首页，并且不能转为pdf，就想在这里用soup</span></span><br><span class="line">        soup = BeautifulSoup(html, <span class="string">&#x27;lxml&#x27;</span>)</span><br><span class="line">        <span class="keyword">with</span> <span class="built_in">open</span>(<span class="string">&#x27;debugdir/&#x27;</span> + <span class="built_in">str</span>(name) + <span class="string">&#x27;.css&#x27;</span>, <span class="string">&#x27;w&#x27;</span>) <span class="keyword">as</span> f:</span><br><span class="line">            f.write(<span class="built_in">str</span>(soup))</span><br><span class="line">        <span class="comment">#print(soup)</span></span><br><span class="line">        content = soup.find(<span class="string">&#x27;div&#x27;</span>,<span class="built_in">id</span>=<span class="string">&quot;content_views&quot;</span>,class_=[<span class="string">&quot;markdown_views prism-atom-one-dark&quot;</span>, <span class="string">&quot;markdown_views prism-atom-one-light&quot;</span> , <span class="string">&quot;htmledit_views&quot;</span>]) <span class="comment">#class_=&quot;htmledit_views&quot;</span></span><br><span class="line">        <span class="built_in">print</span>(content)</span><br><span class="line">        <span class="comment"># print(content)</span></span><br><span class="line">        <span class="comment"># print(title, content)</span></span><br><span class="line">        html = html_str.<span class="built_in">format</span>(article=content)</span><br><span class="line">        self.write_content(html, title)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">write_content</span>(<span class="params">self, content, name</span>):</span><br><span class="line"></span><br><span class="line">        <span class="built_in">print</span>(name)</span><br><span class="line"></span><br><span class="line">        html_path = <span class="string">&quot;HTML/&quot;</span> + <span class="built_in">str</span>(self.change_title(name)) + <span class="string">&quot;.html&quot;</span></span><br><span class="line">        pdf_path =<span class="string">&quot;PDF/&quot;</span> + <span class="built_in">str</span>(self.change_title(name))+ <span class="string">&quot;.pdf&quot;</span></span><br><span class="line">        md_path = <span class="string">&quot;MD/&quot;</span> + <span class="built_in">str</span>(self.change_title(name)) + <span class="string">&quot;.md&quot;</span></span><br><span class="line"></span><br><span class="line">        <span class="comment"># 将内容保存为html文件</span></span><br><span class="line">        <span class="keyword">with</span> <span class="built_in">open</span>(html_path, <span class="string">&#x27;w&#x27;</span>,encoding=<span class="string">&quot;utf-8&quot;</span>) <span class="keyword">as</span> f:</span><br><span class="line">            f.write(content)</span><br><span class="line">            <span class="built_in">print</span>(<span class="string">&quot;正在保存&quot;</span>, name, <span class="string">&quot;.html&quot;</span>)</span><br><span class="line"></span><br><span class="line">        <span class="comment"># 将html文件转换成PDF文件</span></span><br><span class="line">        <span class="comment">#config = pdfkit.configuration(wkhtmltopdf=r&#x27;G:\Dev\wkhtmltopdf\bin\wkhtmltopdf.exe&#x27;)</span></span><br><span class="line">        <span class="comment">#pdfkit.from_file(html_path, pdf_path, configuration=config)</span></span><br><span class="line">        <span class="comment">#print(&quot;正在保存&quot;, name, &quot;.pdf&quot;)</span></span><br><span class="line">        <span class="comment"># os.remove(html_path)</span></span><br><span class="line"></span><br><span class="line">        <span class="comment"># 将html文件转换成md格式</span></span><br><span class="line">        html_text = <span class="built_in">open</span>(html_path, <span class="string">&#x27;r&#x27;</span>, encoding=<span class="string">&#x27;utf-8&#x27;</span>).read()</span><br><span class="line">        markdown = html2text.html2text(html_text)</span><br><span class="line">        <span class="keyword">with</span> <span class="built_in">open</span>(md_path, <span class="string">&#x27;w&#x27;</span>, encoding=<span class="string">&#x27;utf-8&#x27;</span>) <span class="keyword">as</span> file:</span><br><span class="line">            file.write(markdown)</span><br><span class="line">            <span class="built_in">print</span>(<span class="string">&quot;正在保存&quot;</span>, name, <span class="string">&quot;.md&quot;</span>)</span><br><span class="line"></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">change_title</span>(<span class="params">self, title</span>):</span><br><span class="line">        mode = re.<span class="built_in">compile</span>(<span class="string">r&#x27;[\\\/\:\?\*\&quot;\&lt;\&gt;\|\!\t]&#x27;</span>)</span><br><span class="line">        new_title = re.sub(mode,<span class="string">&#x27;_&#x27;</span>, title)</span><br><span class="line"></span><br><span class="line">        <span class="keyword">return</span> new_title</span><br><span class="line"></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">start</span>(<span class="params">self</span>):</span><br><span class="line">        <span class="comment">#response = self.send_request(self.url)</span></span><br><span class="line">        <span class="comment">#if response:</span></span><br><span class="line">        <span class="comment">#    self.parse_content(response)</span></span><br><span class="line">        <span class="keyword">for</span> i, link <span class="keyword">in</span> <span class="built_in">enumerate</span>(self.lst):</span><br><span class="line">            <span class="built_in">print</span>(i + <span class="number">1</span>, link)</span><br><span class="line">            response = self.send_request(link)</span><br><span class="line">            <span class="keyword">if</span> response:</span><br><span class="line">                 self.parse_detail(response, i)</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">&#x27;__main__&#x27;</span>:</span><br><span class="line">    csdn = CSDNSpider(lst)</span><br><span class="line">    csdn.start()</span><br><span class="line"></span><br><span class="line"></span><br></pre></td></tr></table></figure>
<h2 id="3效果"><a class="markdownIt-Anchor" href="#3效果"></a> 3.效果</h2>
<p>虽然得到了所有博客的md文件，整体效果稍微差一点，比如代码框没有代码类型，代码框中回车较多会被截成好几份等，但是也可以接受</p>

      
    </div>
    <footer class="article-footer">
      <a data-url="http://example.com/2022/05/06/%E7%88%AC%E5%8F%96%E8%87%AA%E5%B7%B1CSDN%E4%B8%8A%E6%89%80%E6%9C%89%E7%9A%84%E5%8D%9A%E5%AE%A2%E5%B9%B6%E8%BD%AC%E5%AD%98%E4%B8%BAmd%E6%96%87%E4%BB%B6/" data-id="cl2uhoedl005ke4j36ee38d2w" class="article-share-link">分享</a>
      
      
  <ul class="article-tag-list" itemprop="keywords"><li class="article-tag-list-item"><a class="article-tag-list-link" href="/tags/%E7%88%AC%E8%99%AB/" rel="tag">爬虫</a></li></ul>

    </footer>
  </div>
  
    
<nav id="article-nav">
  
    <a href="/2022/06/06/linux%E6%9C%8D%E5%8A%A1%E5%99%A8%E4%B8%8A%E9%80%9A%E8%BF%87python%E8%84%9A%E6%9C%AC%E8%87%AA%E5%8A%A8%E6%8E%A7%E5%88%B6mysql%E6%95%B0%E6%8D%AE%E5%BA%93/" id="article-nav-newer" class="article-nav-link-wrap">
      <strong class="article-nav-caption">上一篇</strong>
      <div class="article-nav-title">
        
          python语言实现服务器对mysql数据库的自动处理
        
      </div>
    </a>
  
  
    <a href="/2022/05/06/hexo%E6%A1%86%E6%9E%B6%E4%B8%8B%E5%8D%9A%E5%AE%A2%E5%BA%94%E7%94%A8%E5%A4%8F%E5%A8%9C%E4%B8%BB%E9%A2%98%E5%B9%B6%E7%95%A5%E5%BE%AE%E9%AD%94%E6%94%B9/" id="article-nav-older" class="article-nav-link-wrap">
      <strong class="article-nav-caption">下一篇</strong>
      <div class="article-nav-title">hexo框架下博客应用夏娜主题并略微魔改</div>
    </a>
  
</nav>

  
</article>



</section>
        
          <aside id="sidebar">
  
    <div class="widget-wrap">
  <h3 class="widget-title"></h3>
  <div class="widget">
    <h1 class="blog-title">蓝湖畔淅淅沥沥的雨</h1>
    <h2 class="blog-subtitle">All tragedy erased, I see only wonders...</h2>
    <ul class="blog-link">
     
          <a href="/" title="Home">
            <li>主页</li>
          </a>
        
          <a href="/archives" title="Archives">
            <li>归档</li>
          </a>
        
          <a href="/categories" title="Categories">
            <li>分类</li>
          </a>
        
          <a href="/tags" title="Tags">
            <li>标签</li>
          </a>
        
          <a href="/knightabout" title="Knightabout">
            <li>关于</li>
          </a>
        
          <a href="/bridges" title="Bridges">
            <li>传送门</li>
          </a>
        
          <a href="/announcement" title="Announcement">
            <li>公告</li>
          </a>
        
    </ul>
  </div>
</div>

  
    <div class="widget-wrap">
  <h3 class="widget-title"></h3>
  <div class="widget">
    <img class="avatar" src="/images/avatar.png">
    <h2 class="author">StarsWhisper</h2>
    <h3 class="description"></h3>
    <div class="count-box">
      <a href="/archives"><div><strong>75</strong><br>文章</div></a>
      <a href="/categories"><div><strong>31</strong><br>分类</div></a>
      <a href="/tags"><div><strong>56</strong><br>标签</div></a>
    </div>



    <div class="social-link">
      
        <a class="hvr-bounce-in" href="https://github.com/Wldcmzy" target="_blank" title="Github">
          Github
        </a>
      
        <a class="hvr-bounce-in" href="https://blog.csdn.net/wldcmzy" target="_blank" title="CSDN">
          CSDN
        </a>
      
        <a class="hvr-bounce-in" href="https://space.bilibili.com/83743701" target="_blank" title="bilibili(无技术和学习内容)">
          bilibili(无技术和学习内容)
        </a>
      
    </div>

    <div class="friend-link">
      <h2>友情链接</h2>
      
        <a class="hvr-bounce-in" href="https://shanamaid.github.io/" target="_blank" title="夏娜主题作者的博客">
          夏娜主题作者的博客
        </a>
      
    </div>
  </div>
</div>

  
</aside>
        
      </div>
      <footer id="footer">
  
  <div class="outer">
    <div id="footer-info" class="inner">
      &copy;2021 - 2022 StarsWhisper<br>
      由<a href="http://hexo.io/" target="_blank">Hexo</a>强力驱动 | 
      主题-<a target="_blank" rel="noopener" href="https://github.com/ShanaMaid/hexo-theme-shana">Shana</a>(但是魔改)
      
    </div>
    
  </div>
</footer>
    </div>
    

<script src="//apps.bdimg.com/libs/jquery/2.1.4/jquery.min.js"></script>
<script src="//apps.bdimg.com/libs/wow/0.1.6/wow.min.js"></script>
<script>
new WOW().init();
</script>   


  
<link rel="stylesheet" href="/plugin/fancybox/jquery.fancybox.css">

  
<script src="/plugin/fancybox/jquery.fancybox.pack.js"></script>




  
<link rel="stylesheet" href="/plugin/galmenu/GalMenu.css">

  
<script src="/plugin/galmenu/GalMenu.js"></script>

  <div class="GalMenu GalDropDown">
      <div class="circle" id="gal">
        <div class="ring">
          
            <a href="/announcement" title="" class="menuItem">公告</a>
          
            <a href="/tags" title="" class="menuItem">标签</a>
          
            <a href="/categories" title="" class="menuItem">分类</a>
          
            <a href="/archives" title="" class="menuItem">归档</a>
          
            <a href="/knightabout" title="" class="menuItem">关于</a>
          
            <a href="/bridges" title="" class="menuItem">传送门</a>
          
        </div>
        
          <audio id="audio" src="#"></audio>
        
      </div> 
</div>
<div id="overlay" style="opacity: 1; cursor: pointer;"></div>
  <script type="text/javascript">var items = document.querySelectorAll('.menuItem');
    for (var i = 0,
    l = items.length; i < l; i++) {
      items[i].style.left = (50 - 35 * Math.cos( - 0.5 * Math.PI - 2 * (1 / l) * i * Math.PI)).toFixed(4) + "%";
      items[i].style.top = (50 + 35 * Math.sin( - 0.5 * Math.PI - 2 * (1 / l) * i * Math.PI)).toFixed(4) + "%"
    }</script>
<script type="text/javascript">
  $(document).ready(function() {
    $('body').GalMenu({
      'menu': 'GalDropDown'
    })
  });
</script>

  <section class="hidden-xs"> 
  <ul class="cb-slideshow"> 
    <li><span>苟利</span></li> 
    <li><span>国家</span></li> 
    <li><span>生死以</span></li> 
    <li><span>岂能</span></li> 
    <li><span>祸福</span></li> 
    <li><span>趋避之</span></li> 
  </ul>
</section>

<script src="/js/script.js"></script>




  </div>
</body>
</html>